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ABSTRACT 

In  this  paper  we  describe  a  real  time  algorithm  for  a  missile-based  IR  tracking  system.  A  correlation-based  tracker  is 
considered  to  be  the  primary  tracker,  while  a  feature-based  tracker  is  chosen  to  provide  support  on  a  secondary  level.  A 
prescreener  runs  concurrently  with  the  trackers  to  provide  a  list  of  probable  target  centroids.  This  information  is  used  to 
provide  a  level  of  confidence  to  the  tracker.  The  tracker  subfunctions  are  implemented  in  both  programmable  logic  hardware 
and  software  operating  on  digital  signal  processors.  The  maximum  sustained  pixel  data  rate  that  can  be  processed  is  10  MHz, 
which  will  provide  a  60  Hz  tracker  with  a  256  x  256  input  image  and  a  30  Hz  tracker  for  a  512  x  512  input  image.  The 
prescreener  function  is  implemented  in  reprogrammable  hardware  subfunctions  that  operate  up  to  the  full  10  MHz  pixel  rate 
producing  target  centroids. 
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1.  INTRODUCTION 

This  paper  will  discuss  both  the  algorithm  components  and  the  hardware  capabilities  required  to  implement  a  multi-stage 
infrared  (IR)  missile  tracker  in  real  time.  In  practice,  there  are  several  issues  that  need  to  be  considered  when  developing  an 
imaging  infrared  missile  tracker.  The  detection  and  tracking  algorithms  are  required  to  be  robust  to  changes  in  the  target 
signature  which  will  vary  depending  on  the  time  of  day,  environment,  and  atmospheric  conditions,  as  well  as  aspect  and 
depression  angles.  In  addition,  the  number  of  pixels  on  target  will  vary  dramatically  from  the  first  to  last  frame  of  the  missile 
flight. 


2.  TRACKER  OVERVIEW 

This  tracker  algorithm  has  two  main  components.  The  first,  known  as  the  primary  tracker,  consists  of  a  correlation  tracker. 
The  correlation  tracker  begins  tracking  on  a  predesignated  target.  In  the  event  that  the  primary  tracker  does  not  meet  a 
predetermined  set  of  conditions,  the  secondary  tracker  takes  over.  The  correlation  tracker  is  implemented  in  multiple  digital 
signal  processors  (DSPs)  utilizing  a  dedicated  frame  memory  containing  both  intensity  and  edge  image  data  for  each  entire 
frame.  These  frame  memories  are  updated  at  the  full  60  Hz  frame  rate  and  can  be  addressed  by  any  of  the  DSPs.  The 
secondary  tracker  is  a  feature -based  tracker  in  which  segments  corresponding  to  each  target  centroid  are  extracted  from  the 
IR  image  and  various  features  associated  with  them  are  calculated.  The  features  are  input  into  a  similarity  metric  that 
incorporates  target  statistics  acquired  during  the  primary  tracker  mode.  A  high  degree  of  similarity  ensures  a  valid  target 
track.  Once  the  similarity  measure  reaches  a  predetermined  threshold,  the  primary  tracker  is  again  enabled.  The  feature- 
based  tracker  utilizes  a  single  DSP  in  conjunction  with  the  image  frame  memories  to  compute  and  analyze  the  desired 
features  at  a  reduced  data  rate.  Image  feature  statistics  will  be  kept  on  the  tracked  object  locally  within  the  single  DSPs  local 
memory.  Figure  2  shows  the  tracker  hardware  block  diagram. 
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Figure  1  Tracker  Components 
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Figure  2  Tracker  hardware  Diagram 


3.  CORRELATION  TRACKER 


The  correlation  tracker  is  the  primary  tracker  of  the  system.  It  calculates  the  minimum  mean  square  error3  (MMSE)  between 
a  rectangular  window  of  pixels  referred  to  as  the  reference  window  and  another  larger  rectangular  region  of  search  window 
pixels  as  shown  in  Figs  3  and  4.  The  tracker  hardware  will  support  up  to  a  128  x  128  search  window  and  64  x  64  reference 
window  sizes  at  the  full  60  Hz  frame  rate.  The  correlation  coefficient  corresponding  to  the  pixel  location  of  the  minimum  of 
the  MMSE  surface  is  calculated  and  provides  a  measure  of  the  validity  of  the  track. 


MMSE(x,y)  =  min 


^  f  current  (x,y)  -  8history(x  +  X,y  +  y) 
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f  current  (•*>  j)  =  the  current  search  window  gated  by  a  mask  of  ones  the  size  of  ghistory 
and 

8 history  (A}7)  =  the  reference  window  used  in  the  correlation. 


The  first  term  in  the  above  equation  is  referred  to  as  the  “correction  ternT'  and  is  essentially  a  normalization  term.  The 
middle  term  is  referred  to  as  the  correlation  term  while  the  last  term  is  a  constant. 

Since  correlation  in  the  spatial  domain  is  essentially  equivalent  to  multiplication  in  the  frequency  domain,  it  is  often  more 
efficient  to  process  in  the  frequency  domain  using  FFTs.  So  the  MSE  can  be  computed  in  the  frequency  domain  as: 

MSE(wl,w2 )  =  F2 (wvw2)M* (w1,w2  )  —  2F(wl, w2)G*(wl, w2)  +  G2(wl,w2) . 


The  first  reference  window  is  obtained  by  extracting  a  region  of  the  image  that  is  centered  on  the  target  and  just  large  enough 
to  encompass  the  entire  target.  The  sizes  of  the  search  window  and  the  reference  window  will  vary  frame  to  frame  depending 
on  the  range  to  the  target.  The  search  window  is  defined  to  be  a  rectangular  region  surrounding  the  target  that  is  at  least  twice 
as  large  as  the  reference  window. 

Once  the  minimum  location  corresponding  to  the  target  is  obtained,  its  coordinates  are  sent  to  the  guidance  and  control  unit 
and  the  gimbal  is  slewed  to  that  location. 


Since  the  target  is  constantly  changing  contrast  due  to  aspect  and  depression  angle  changes,  the  reference  window  being  used 
at  each  frame  is  actually  a  lag  filtered  version  of  the  previous  reference  window  and  the  current  window.  Assume  that  ghistory 
refers  to  the  reference  window  being  used  in  the  /th  frame  correlation  calculation,  the  equation  for  determining  it  is  given  by 

Shistory  =  X*8  current  +  (1  —  x)*ghis,ory 


i-i 


where 

ghistory  =  the  weighted  reference  window  at  frame  /; 

i 


8 current  =  a  reference  window  size  region  centered  on  the  target  extracted  from  the  current  frame  i, 

i 

ghistory  =  the  reference  window  used  for  correlation  in  the  previous  frame  i- 1, 

i- 1 

and 


X  =  the  reference  window  update  rate  (<1). 


If  the  correlation  value  of  the  corresponding  MMSE  drops  below  a  given  threshold,  Tl,  for  Ml  out  of  N1  times,  the  value  of 
x  is  increased  to  account  for  a  target  signature  change,  possibly  due  to  turning.  This  incorporates  the  new  target  signature 
information  into  the  reference  window.  It  is  not  increased  too  quickly  in  case  the  target  is  temporarily  occluded  by  a  slight 
obscuration. 

If  the  correlation  value  of  the  corresponding  MMSE  drops  below  a  given  threshold,  T2,  for  M2  out  of  N2  times,  the  tracker 
switches  to  the  feature -based  tracker. 
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Figure  3  Correlation  Windows 


Figure  4  Example  Correlation  Regions 


4.  FEATURE-BASED  TRACKER 


The  feature -based  tracker  operates  on  the  idea  that  if  the  correlation  tracker  confidence  drops  due  to  some  artifact  in  the 
imagery  or  minor  obstruction,  the  general  features  of  the  target  will  still  be  present  and  can  be  used  to  form  a  measure  of  the 
track  validity.  While  the  correlation  tracker  is  operating,  features  of  segments  associated  with  target  detections  provides  by 
the  prescreener  (which  is  discussed  in  the  next  section)  are  computed.  The  moving  mean  and  variance  of  these  features  over 
P  frames  are  calculated  and  stored  for  later  use  in  a  similarity  measure.  The  Mahalanobis  distance  metric1  is  used  to 
determine  the  degree  of  similarity  of  a  current  target  segment  to  past  observations  of  the  target  being  tracked.  In  this 
formulation,  it  is  assumed  that  the  chosen  features  are  essentially  uncorrelated  and  thus  a  diagonal  matrix  containing  the 
inverse  variance  of  each  of  the  features  replaces  the  inverse  covariance  matrix.  The  similarity  metric  is  given  by 

M  =  Mahalanobi  s  distance 
W;  =  weighting  coefficien  t  on  ith  tracki  ng  feature 
X;  =  value  of  ith  tracki  ng  feature  for  current  cycle 
X;  =  mean  of  ith  tracki  ng  feature 
8  f  =  variance  of  ith  tracki  ng  feature 


M 


In', 

/= 1 


If  there  is  one  detection  provided  by  the  prescreener  corresponding  to  the  search  window  region,  the  Mahalanobis  distance  is 
calculated.  If  it  is  smaller  than  T2  for  M3  out  of  M4  times,  the  target  corresponding  to  the  detection  is  considered  to  be  a 
valid  track  and  its  pixel  coordinates  are  sent  to  the  GEU  and  the  tracker  returns  to  the  primary  mode  of  tracking. 

5.  PRESCREENER 

The  prescreener  is  the  missile  target  detector.  Its  purpose  is  to  provide  a  list  of  all  possible  target  centroids  in  each  image 
frame.  Range  to  the  center  of  the  infrared  image  prior  to  launch  is  assumed  known.  From  this,  it  is  possible  to  calculate  range 
for  every  line  in  the  image  and  in  turn  the  appropriate  number  of  pixels  corresponding  to  a  target  at  any  location  in  the  image. 
Hardware  FPGAs  are  used  to  create  Sobel2'3  edge  image  magnitude  and  direction  images  from  the  input  video  stream  in  real¬ 
time.  These  FPGAs  can  be  programmed  to  apply  any  user  defined  3x3  operator.  Reprogrammable  hardware  is  available  to 
calculate  real-time  histogram  data  about  various  images  within  the  prescreener  process.  The  histogram  data  is  placed  in  FIFO 
memories  that  are  read  by  the  master  DSP  processor  to  compute  image  threshold  levels.  The  first  step  in  the  prescreener 
calculates  strong  edges  in  the  image  by  applying  the  Sobel  edge  operator.  Once  the  Sobel  edge  magnitude  is  calculated,  it  is 
thresholded  and  binarized.  The  thresholded  value  is  determined  from  the  Sobel  magnitude  histogram.  It  is  chosen  by 
determining  which  value  corresponds  to  a  given  slope  of  the  histogram  starting  from  the  maximum  bin  and  working  towards 
the  lowest.  The  image  is  then  integrated  with  variable  box  sizes,  from  a  minimum  of  3  x  3  to  a  maximum  of  64  x  64, 
corresponding  to  possible  targets.  It  is  again  thresholded  according  to  the  histogram.  Once  the  integrated  image  has  been 
thresholded,  it  is  binarized  and  the  target  centroids  are  calculated  by  applying  the  morphological  erode  operator.  The  number 
of  erodes  necessary  to  create  each  centroid  is  stored  and  compared  to  the  number  that  would  be  necessary  to  erode  a  target  at 
the  given  location  in  the  image.  If  it  is  too  large  or  too  small,  the  object  is  considered  to  be  invalid  and  that  centroid  value  is 
discarded.  The  remaining  centroids  comprise  the  detection  list  for  the  current  image  frame.  Each  of  the  prescreener 
functions  have  been  implemented  in  hardware  FPGAs  to  allow  the  prescreener  algorithm  to  operate  at  real-time  frame  rates; 
producing  a  target  candidate  list  for  each  frame  of  input  image  data.  Images  corresponding  to  the  above  steps  are  shown  in 
Figs.  5-8. 
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Figure  5  Prescreener  Flow 


Figure  6  Input  Image 


6.  CONCLUSIONS 


This  paper  has  outlined  the  algorithm  and  hardware  components  that  comprise  a  real  time  missile-based  imaging  infrared 
tracker.  Hardware  and  software  implementations  of  the  tracker  concepts  have  matured  to  a  level  where  developed  algorithms 
have  been  implemented  in  hardware  to  minimize  the  required  software  and  to  reduce  processor  load  requirements.  Due  to  the 
processing  power  and  versatility  of  current  state-of-the-art  DSP  chips,  some  of  the  tracker  algorithms  (i.e.,  FFT,  correlation) 
meet  the  overall  tracker  mode  timeline  requirements  as  software  components  as  opposed  to  dedicated  hardware.  The  tracker 
concepts  have  been  packaged  in  both  tactical  missile  guidance  flight  hardware  as  well  as  standard  laboratory  electronic  card 
formats. 


REFERENCES 

1.  R.  Duda  and  P.  Hart,  Pattern  Classification  and  Scene  Analysis,  John  Wiley  &  Sons  Inc.,  New  York,  1973. 

2.  R.  Gonzalez  and  P.  Wintz,  Digital  Image  Processing  Second  Edition,  Addison-Wesley,  Massachusetts,  1987. 

3.  W.  Pratt,  Digital  Image  Processing  Second  Edition,  John  Wiley  &  Sons  Inc.,  New  York,  1991. 


